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© Multimedia frame relay codec. 



© A multimedia codec for a frame relay network. 
Multiple information media from different sight equip- 
ment is converted to a standard packet for transmis- 
sion over said frame relay network. Incoming frames 
are processed on a priority basis and stored in an 
elastic buffer. Tests of pointers are kept for each 



media type identifying the location in the elastic 
buffer of the various packets for the different media. 
Applications running in the frame relay processor 
retrieve the stored media data and forward it to a 
digital signal processor connected to respective ter- 
minal equipment. 
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The present invention relates to public or pri- 
vate frame relay networks. Specifically, a multi- 
media codec for converting voice, fax and/or video 
information into frame relay packets and decoding 
received frame relay packets into voice, fax and/or 
video data is described. 

Frame relay wide area networks are now in use 
internationally. These networks provide for the dis- 
tribution of data packets between a sending loca- 
tion and a receiving location. The frame relay net- 
work provides for variable time frames for routing 
packets to a specified destination location. 

The frame relay network being all digital holds 
the possibility of sending all media information 
which can be reduced to a packetized digital sig- 
nal. In the case of voice transmission coding al- 
gorithms may be used to maximize the bandwidth. 
Further, techniques used in related technology, 
such as cellular telephone communication, which 
relies on voice-activation for transmission of a 
packet, silent periods between speech bursts are 
available for other packet transmission, may be 
implemented in the frame relay network environ- 
ment. 

The frame relay network presents problems 
which are unique to frame relay networks. For 
instance, the network has a delay which must be 
accommodated at each codec, both transmitting 
and receiving. Further, the algorithm used to en- 
code any of the media signals must be tolerant to 
the occasional frame loss occurring in a frame 
relay network. Forward frame loss correction must 
also be provided, compatible with the various me- 
dia information carried over the frame relay net- 
work. 

The implementation of a multimedia network 
requires that there be a capability for processing 
incoming frames from the frame relay network so 
that data packets belonging to one media, such as 
voice, are distinguishable from data packets which 
may be identified with a video service or data 
service. Further, there must be a priority mecha- 
nism employed in any frame relay interface which 
will handle incoming data packets on a priority 
basis. 

In order to make use of the standard media 
hardware, it is important that the codec operate 
with standard voice data, fax data and/or video data 
formats. Thus, support must be given to 56 kbps 
audio, 9.6 kbps fax signals, as well as video inter- 
faces for the recognized 174 kbps and 384 kbps 
digital video. 

Summary of the Invention 

A multimedia frame relay codec is provided 
which can service incoming frames received on a 
node of the frame relay network on a priority basis, 



while preparing outgoing frames on a background 
basis. The codec interfaces with line interface units 
of different signal media such as digital voice, fax 
and video signals. A digital signal processor asso- 

5 ciated with each of the line units converts the 
received signals into compressed data formats us- 
ing any one of a number of conventional compres- 
sion algorithms. 

A transport processor provides for the interface 

10 between the digital signal processors and node of 
the frame relay network. The frame relay processor 
operates in the foreground to process frame bytes, 
stripping and sending measurement bytes to a 
frame relay statistical application running in the 

75 frame relay processor. Voice data, fax data and 
video data are moved from the frame and loaded 
into an elastic buffer. 

The frame relay processor also creates an out- 
going frame comprising one or more streams of 

20 data received from the digital signal processors. 
Overhead bytes are added to the frame, defining a 
sequence number for each frame which is used in 
the frame relay application of the receiving codec. 
The frame relay processor is organized into 

25 foreground and background operations. Foreground 
operations having priority will process incoming 
frames by removing data contained therein and 
storing for each of the various media data packets 
in a common elastic buffer. Tables of lists are 

30 compiled, identifying the location in the elastic buff- 
er of data packets pertaining to a specific media. 
Application programs operating in the background 
will process the stored data and forward it to the 
respective digital signal processor and its con- 

35 nected line unit. 

Description of the Figures 

Figure 1 is an overall block diagram of a mul- 
40 timedia frame relay codec in accordance with the 
present invention. 

Figure 2 is a functional block diagram of the 
multimedia frame relay codec of Figure 1 . 

Figure 3 illustrates the protocol layers used for 
45 communicating between multimedia frame relay 
codecs. 

Figure 4A illustrates the format of a frame used 
on a frame relay transmission line carrying mul- 
timedia data. 

so Figure 4B illustrates the frame transfer between 
the DSP and frame relay processor of the codec. 

Figure 5 shows the power-up sequence for the 
frame relay codec. 

Figure 6A shows a voice call set up for initiat- 
55 ing voice traffic over the frame relay. 

Figure 68 is a flow chart illustrating how the 
digital signal processor processes audio samples 
received from the line interface unit and frame 
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relay processor. 

Figure 6C shows the data transfer between the 
digital signal processor and frame relay processor 
for other media applications. 

Figure 7 illustrates the frame relay codec setup 
of a fax call between codecs. 

Figure 8 illustrates a video connection setup 
between frame relay codecs. 

Figure 9 is a flow chart illustrating the transfer 
of multimedia information over the frame relay. 

Figure 10 illustrates the call disconnect pro- 
cess for disconnecting from the network node of a 
frame relay network. 

Figure 11 illustrates the processing of mul- 
timedia data received over a frame relay. 

Description of the Preferred Embodiment 

Referring now to Figure 1, there is shown a 
frame relay network 9 and a node of the network 8. 
The node 8 is a port on the network to which a 
codec can be attached for receiving and transfer- 
ring information over the frame relay network 9. 

The network 8 is connected to a codec through 
a line interface unit 10. The interface unit provides 
for termination functions to the node 8 and codec. 
The line interface functions are more particularly 
described in an industry standard such as CCITT 
V35 EIA 530. 

The process of creating frames for transmis- 
sion on the flame relay network 9 is carried out in a 
frame relay processor 1 1 . The frame relay proces- 
sor will set up calls over the frame relay network 9, 
as well as remove data from a frame on the frame 
relay network 9, addressed to the codec. The 
frame relay processor receives from a plurality of 
digital signal processors 12-16, packets of digital 
data which may belong to different media, such as 
digitized audio, digitized fax, digitized image, 
digitized video or digitized computer terminal data. 
The packets received from each of the digital sig- 
nal processors 12-16 are formatted in accordance 
with the frame relay network protocol, and a frame 
is constructed including the various media data 
packets and the required frame overhead packets 
for transferring the data over the frame relay net- 
work. 

Additionally, packets received over the frame 
relay network 9 are organized by the frame relay 
processor 1 1 into distinct media types, so that they 
can be conveyed to the appropriate digital signal 
processor 12-16. Each of the digital signal proces- 
sors 12-16, as well as a separate line carrying 
computer data from a line interface unit 25, is 
connected to an input/output port of the frame relay 
processor. The frame relay processor 11 will con- 
duct a data transfer for packets received over the 
frame relay network 9 to a corresponding digital 



signal processor 12-16. 

The respective digital signal processor per- 
forms signal processing functions such as encod- 
ing or decoding for multimedia signals received 

s from a respective line interface unit 21-25 The 
digital signal processors will employ standard com- 
pression algorithms and convert the input signals to 
data packets which will be forwarded to the frame 
relay processor 11. The frame relay processor 11 

10 may be a standard processor such as the Intel 
486/25 MHz. processor configured as will be de- 
scribed to operate the frame relay network pro- 
tocol. 

Each of the digital signal processors also por- 
ts forms the call processing between terminal equip- 
ment connected to each of the line interface units 
21-25 and the frame relay processor, which will 
receive incoming frames of media data for each of 
the terminal equipment. The digital signal proces- 
20 sors may be of a standard design, such as the 
Texas Instruments TMS 320C30. 

As illustrated in Figure 1, where computer ter- 
minal data equipment is provided no digital signal 
processor is necessary, and the respective line 
25 interface unit maybe connected directly to a in- 
put/output port of the frame relay processor 1 1 . 

The operation of the codec of Figure 1 is 
functionally illustrated in the block diagram of Fig- 
ure 2. Turning now to Figure 2, each of the media 
30 terminal input lines is shown connected to respec- 
tive line interface functions 31 through 34. The line 
interfaces provide for impedance matching be- 
tween the respective terminals and the following 
digital processing equipment. 
35 The DSP functions 41 through 43 are unique to 

the specific media which they process. The DSP 
function 41 for voice signal processing provides for 
the speech compression/decompression using 
standard compression algorithms. The communica- 
40 tion path between the terminal equipment and the 
digital signal processing is full duplex, permitting 
the transmission and reception of data to the re- 
spective terminals. 

The voice signal processing function 41 pro- 
45 vides for speech frame buffering and packetization 
of incoming data packets. Further, when a voice 
terminal goes off hook, the signalling packets are 
generated for initiating a call set up over the frame 
relay network. The usual protocol functions in a 
so voice communication path, i.e., detection of on/off 
hook, the detection of dial digits or tones, is imple- 
mented in the voice signalling processing function 
41. Further, the aforesaid digital signal processor 
may implement error correction. 
55 Signal processing function 42 provides for a 
fax interface to the frame relay processor. The fax 
signal processing function 42 receives data from a 
respectively connected fax terminal as a two-way 
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communication path through the A-D/D-A converter 
function 37. Fax signal processing is not full du- 
plex, as the transmit receive functions do not occur 
simultaneously. 

The fax demodulation using standard modem 
signalling is provided by the fax signal processing 
function 42. The signal processing function may 
implement the CCITT V29, V27 protocol for creat- 
ing packets of digital data representing the fax 
transmission. Further, received fax transmission 
packets over the frame relay network are decoded 
and applied to the digital to analog function 37. 

The signal processing function 42 will provide 
the data/message frame buffering/packetization, as 
well as synchronization between the frame relay 
processor and the connected fax terminal equip- 
ment. As in voice signal processing, phone line 
handling, i.e., on/off hook status, dial digits, etc., 
are decoded in the signal processing function 42, 
and forwarded to the frame relay processor for 
setting up the connection to the dial destination. 

As a further example of the multimedia aspect 
of the present invention, video image information 
may also be transferred via the frame relay net- 
work. Trie line interface unit 33 provides for the 
required interface/ impedance matching to the ana- 
log/digital and digital/analog circuits presented by 
function 38. A video signal processing function 43 
is implemented by the respective digital signal 
processor for providing video compression/ decom- 
pression through standard compression algorithms. 
The data is packetized and synchronized to the 
incoming scan rate. 

All the signal processing functions 41 through 
43 provide a frame relay processor interface func- 
tion. The input/output ports connecting the digital 
signal processors to the frame relay processors 
receive and transfer data under control of the frame 
relay processor. The frame relay processor func- 
tions are illustrated as Items 50 through 59. Each 
of the media acquire an application for processing 
received and transmitted media data. The voice 
application layer 50, fax application processing 51, 
video application layer processing 52 and data 
MUX/DEMUX processing 53 format the packets of 
media data for transmission in the frame of a frame 
relay network. There is a transport layer processing 
54 which provides for the common end-to-end de- 
livery and session control functions for frames on a 
frame relay network. 

Frames of multimedia data created by the ap- 
plication layer processing 50-53 are stored in a 
transmit queue 56, along with the transport layer 
information. Incoming data received over the frame 
relay network is processed in a frame relay layer 
processing stem 58 to remove the various headers 
associated with the frame relay and to store recov- 
ered media data in an elastic buffer 57. 



The application layer processing 50-53 and 
transport layer processing 54 operate in the back- 
ground of the frame relay processor. The fore- 
ground processing is devoted to processing incom- 

5 ing frames of multimedia data. 

The multimedia frame relay codec employs 
protocol layer processing to create frames of mul- 
timedia data as well as to process incoming frames 
of multimedia data. 

w Turning now to Figure 3, the process of pro- 
tocol layering carried out by the frame relay pro- 
cessor is shown. The topmost layer is the applica- 
tion layer provides specific signal processing of the 
multimedia traffic. This would include the call pro- 

15 cessing initiated by users, the processing afforded 
by the digital signal processor of each media type. 
Creation of the various packets for transmission 
over the frame relay network occurs within the 
application layer operating in the background of the 

20 frame relay processor. 

Trie multimedia transport layer provides for 
common end-to-end data delivery and session con- 
trol functions. This would include sequencing, de- 
lay compensation and network error handling for 

25 the frames transmitted and received over the frame 
relay network. 

The standard frame relay network signalling 
functions are carried in the frame relay network 
layer. The frame relay network layer provides for 

30 network management of each of the transmitted 
frames. 

A frame relay link layer provides the standard 
relay network core link layer functions. These 
would include the framing of the headers, media 
35 data packets and synchronization functions which 
are common to the frames sent over a frame relay 
network. 

The bottom protocol layer is the physical layer. 
This is the physical connection to the frame relay 
40 network node. The requirements for this layer are 
set forth in the standards, such as CCITT V35/EIA 
530. 

Trie physical layer frame relay fink layer and 
frame relay network layer are all standard frame 

45 relay protocols and are described in the ANSI T- 
617 document. 

The foregoing protocol processing produces a 
frame of data bits in accordance with Figure 4A. 
Figure 4A shows the distinct fields of the frame. 

so The line idle portion contains the standard frame 
relay idle pattern of 01111111. The frame relay 
header also contains standard frame relay header 
information such as the network connection iden- 
tifier which is addressing information for the net- 

55 work (DLCI) and other network control information. 
This would include the congestion and priority in- 
formation contained in a standard frame relay 
header. 
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Adapting the frame relay network for use in the 
multimedia environment requires a multimedia 
transport header. This header contains information 
which is sent transparently across the frame relay 
network to provide common multimedia transport 
functions for the receiving codecs. The fields con- 
stituting the multimedia transport header include 
the following options: 

1) Protocol Identifier (for example, a multimedia 
or other standard frame relay protocol 

2) Message Type (i.e., the transport setup in- 
formation acknowledgement and connection sta- 
tus) 

3) Error Recovery Option 

4) Multi-Cast Connection Option 

5) Message Transmit Sequence Number 

6) Message Receive Sequence Number Option 
and a Delay Measurement Variable. 

A multimedia application header is also con- 
tained within the framework. The multimedia ap- 
plication header contains more information, sent 
transparently across the frame relay network, which 
is needed by the voice/fax/video data applications 
running/in the frame relay processor of the codec. 
This includes fields which have an application iden- 
tifier to identify voice, fax, video data or other 
media formats, as well as a message type to 
identify information packets from control packets to 
each of the receiving codecs. 

The field identified as application information 
contains the media packets which were generated 
by the digital signal processing devices of each 
transmitting codec. These packets can be voice, 
fax, video or data bits, or alternatively, user call 
control messages. The application header will de- 
fine to which media type the particular data packet 
belongs. 

Finally, the frame concludes with an error de- 
tection code to permit each of the nodes to do 
error checking. 

As illustrated in the foregoing frame, the ap- 
plication running in the frame relay processor cre- 
ates a single frame containing data bits belonging 
to one or more multimedia signals. For example, 
video and audio information may be sent in the 
same frame during a common audio/video session 
application. Speech and/or graphics information 
may be sent in the same frame for a common 
audio graphics application. The sharing of data is 
utilized when needed for synchronization and band- 
width saving purposes. 

No matter what the media being sent over the 
frame relay network, the end-to-end transport pro- 
tocol is common to all media and the protocol 
elements such as sequencing, activity detection, 
network delay compensation, synchronization, error 
handling and point-to-point/multicast connectivity 
are the same. The transport and application layer 



protocols are specifically designed for the mul- 
timedia applications over frame relay networks. 

The digital signal processors convert all the 
different media information signals into a common 

5 packet size so that they can be handled inter- 
changeably. The digital signal processors may be 
configurable by software, downloaded from a par- 
ticular frame relay processor selected by the codec 
operator. Thus, a single digital signal processor 

70 element might be used for processing voice or 
some other media at the selection of the operator. 

Figure 4B illustrates the time sequence in 
which data is exchanged between the digital signal 
processor and frame relay processor buffer. 

75 The digital signal processor receives the multi- 
media data as a continuous bitstream from the line 
interface unit 21 through 25 of Figure 1 . The digital 
signal processor stores the received data in a 
memory buffer for a sample time which, in the 

20 case of voice media, is 20 milliseconds. Trie digital 
signal processor then encodes the data, through 
standard compression algorithms, and writes the 
compressed output in a memory area which is 
accessible by the frame relay processor. The digi- 

25 tal signal processor sets a SYNC flag in this mem- 
ory area, indicating that the compressed data is 
ready for transfer over the frame relay. 

The frame relay processor reads the com- 
pressed data and creates the frame, including the 

30 compressed audio and any other media data re- 
ceived from other line interface units of other digital 
signal processors. The frame relay processor com- 
pletes processing of the outgoing frame, and then 
resets the SYNC flag on the digital signal proces- 

35 sor. This time, the application running in the digital 
signal processor, be it voice data, fax, etc., checks 
the elastic buffer for any frame received from the 
frame relay line. The frame relay processor builds 
the received frames in its memory, and then writes 

40 the frame to the digital signal processor memory 
area reserved for receiving frames from the frame 
relay processor. At this time, the frame relay pro- 
cessor resets the SYNC flag to zero on the initial 
signal processor. 

45 The sequence of codec processing is illus- 
trated in Figures 5-10. Referring to Figure 5, the 
power-up process begins with the start block 70. 
The frame relay processor, as well as the digital 
signal processors are powered-up in a normal corn- 
so puter processor power-up condition. 

Based on the operator selections entered 
through the frame relay processor, the frame relay 
processor loads the operating program from the 
system disk. During a download routine, each of 

55 the digital signal processors are configured in step 
73 to match the connected terminal equipment. 
The various protocols relating to voice/fax/ video 
are configured as needed in each of the digital 
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signal processors. 

Once the digital signal processors are config- 
ured for their respective connected terminal equip- 
ment, the interface between the frame relay pro- 
cessor and digital signal processors are initialized. 
This interface may be a plurality of serial ports, 
each of which under control of the frame relay 
processor receives and/or delivers data to a re- 
spective connected digital signal processor. 

When the initialization routine is completed in 
step 75, the codec is ready to set up a call for any 
one of the connected terminal equipment. Figure 6 
in particular illustrates a voice call set up when the 
voice terminal equipment goes off-hook. 

As step 80 illustrates, the off-hook is detected 
at the digital signal processor connected to the 
telephone line interface unit. Additionally, a signal 
from the frame relay processor will indicate incom- 
ing traffic for a connected telephone terminal. The 
command to the digital signal processor generates 
a ringing current out to the connected telephone 
terminal. 

The off-hook condition 81 results in the digital 
signal processor sending dial tones back to the 
connected telephone terminal. The digital signal 
processor collects dial digits from the connected 
telephone terminal in step 83, and passes the 
collected digits to the voice application on the 
frame relay processor. 

The frame relay processor in step 84 executes 
the voice application program in the background 
and maps the collected digits to a network address. 
The frame relay processor will begin to create the 
frame for transmission over the frame relay net- 
work, which includes the network and transport 
connections. 

The frame relay processor, having initiated a 
frame containing the network transport connections, 
waits for the frame relay network to return a remote 
phone line connection, indicating that the destina- 
tion party has been reached and has gone off- 
hook. 

Steps 86, 87 and 88 demonstrate the activity at 
the remote destination codec. The frame relay in- 
put, in step 86, carries the dialed connection re- 
quest to the destination codec. When the destina- 
tion codec goes off. hook in 87, a remote response 
is sent through a second frame originated in step 
88 at the remote destination codec over the frame 
relay network. 

The call setup protocol, standard in most 
telephony applications, results in a voice connec- 
tion being made in step 90. 

The process for transferring the voice data 
between a digital signal processor and the frame 
relay processor is shown in a flow chart diagram in 
Figure 6B. 



The digital signal processor waits for each 
sample time to receive the voice data from the line 
interface unit in step 91. The receive data is en- 
coded in step 92, using the conventional data com- 

s pression algorithms. The encoded data is written to 
the digital signal processor memory, and then to 
the frame relay buffer in step 93. The SYNC flag at 
the digital signal processsor is set to 1 in step 94. 
Once the SYNC flag has been set to zero, 

io indicating that the frame relay processor has voice 
data to transfer to the digital signal processor in 
step 95, the digital signal processor will read the 
frame relay buffer data in step 96. The data is 
decoded in step 97 by removing the various head- 

75 ers associated with the data and the data is subject 
to decompression. The recovered data from step 
97 is transmitted to the line interface unit in step 
98. 

In the frame relay processor, various steps are 

20 executed as shown in Figure 6C. When the SYNC 
flag has been set to 1 in step 102, the digital signal 
processor data is read from its respective memory 
to the frame relay processor buffer in step 103. In 
the case of a voice application running in the digital 

25 signal processor, the application checks whether 
the data is speech or silence, based on the signal 
energy level. No further processing is done for 
silent data. For speech data, the frame relay pro- 
cessor readies the frame for transmission by insert- 

30 ing the application in transport layer header in- 
formation, including application identifier and se- 
quence number in the proper memory locations in 
step 105. The frame relay buffer is then banded 
over to the frame relay layer which builds the 

35 remainder of the frame, including the frame relay 
header and CRC code. The completed frame is 
transmitted on the frame relay line in step 1 07. 

Once the frame is transmitted, the voice ap- 
plication in the frame relay processor checks the 

40 elastic buffer in step 108 for any frame data re- 
ceived from the frame relay. The processing of the 
frame contained in the elastic buffer occurs by 
reading the list of pointers created by the frame 
relay processor. When the pointer indicates there 

45 is a new frame in the frame relay buffer, the 
sequence number contained in the header informa- 
tion is consulted to determine whether the frame is 
to be handed to the digital signal processor imme- 
diately or delayed for some time, as would occur 

50 when silence periods occur in voice data. For the 
case where silence is indicated by the frame relay 
header information in step 109, the frame relay 
processor in step 110 will build a frame in its 
memory which contains the appropriate comfort 

55 noise for this conversation. The frame relay proces- 
sor in step 120 then writes the frame into the digital 
signal processor memory area reserved for receiv- 
ing frames from the frame relay processor buffer 
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memory. At this time, in step 121, the SYNC flag is 
set to zero. 

Fax calls are similarly set up and the process 
for setting up such a fax call is shown in Figure 7. 
Referring now to Figure 7, block 130 indicates that 
the digital signal processor which has been initial- 
ized for the fax terminal equipment operation, waits 
for an off-hook condition or a frame relay input 
signal indicating fax traffic has been received. The 
off-hook initiated by a connected terminal fax fa- 
cility in step 131 results in a dial tone being sent 
from the digital signal processor in step 97 back to 
the fax terminal Dial digits are initiated at the fax 
terminal and are collected at the digital signal pro- 
cessor. The digital signal processor will pass the 
collected digits onto the frame relay processor for 
setting up the connection between the originating 
codec and the destination codec. 

The frame relay processor initiates the frame 
for setting up the connection in step 101. The 
destination codec goes off-hook in step 1 02 when it 
receives the request from its node connected to 
the frame relay network The off-hook condition is 
reported via a frame from the destination node to 
the originating node, indicating a connection has 
been set up. 

Step 105 illustrates the normal fax application 
tone exchange which occurs between the connect- 
ed codecs. Each of the digital signal processors 
will initiate an exchange of the required fax tones 
per the CCITT standard and provide for modulation 
and demodulation of the fax signal on the required 
carriers and from the carriers. 

A transfer of the image following the appro- 
priate tone exchange between connected fax facili- 
ties occurs in step 106. 

The multimedia codec handles a video connec- 
tion in a similar manner. Referring to Figure 8, the 
video application running in the background of the 
frame relay processor waits for a video signal to be 
applied from a connected digital signal processor 4 
from a frame received on the frame relay network 
in step 109. 

To initiate a video transmission from the codec, 
the operator initiates video transmission in step 
110. The connected digital signal processor will 
detect activity on its connected port, and signal the 
frame relay processor to set up network and trans- 
port functions for a video data transfer. 

The frame relay processor will create the requi- 
site frame having the destination digits and trans- 
port information and apply the same to the frame 
relay network in step 113. An addressed network 
port on the frame relay network will receive and 
decode the received frame. Recognizing the des- 
tination address, the connected frame relay proces- 
sor will initiate activation of its digital signal proces- 
sor in step 114, which is connected to its respec- 



tive video terminal. A response is sent in step 115 
over the frame relay network, via the respective 
remote frame relay processor completing the con- 
nection to the originating codec. 

5 At this point, a video transfer will occur in step 

118 such that video is transmitted via the first 
codec as packetized video data over the frame 
relay network to the destination codec and its con- 
nected video terminal. 

io The information transfer phase which accom- 

panies any one of the frame relay multimedia sig- 
nal transfers is illustrated in Figure 9. The informa- 
tion transfer step 120 begins with information re- 
ceived from the connected multimedia line in step 

is 121. This can be any of the multimedia lines which 
may be receiving traffic once a connection has 
been set up. 

The respective line interface unit performs ana- 
log digital conversion in step 122, where such 

20 media is in the analog format. The digital signal 
processor will perform the signal processing which 
may include, in the case of a voice transmission 
encoding the voice with a compression algorithm, 
as well as packetizing the voice. Further, in the fax 

25 situation, a demodulation may occur of the re- 
ceived fax carriers to obtain a signal which can be 
converted to digital packets. In the case of a video 
transmission, the video is compressed and con- 
verted to digitized packets. 

30 In step 124, the digital signal processor per- 

forms activity detection, detecting the presence of 
such information and the respective packetization 
for sending to the frame relay processor. The 
frame relay processor builds the frame of Figure 4 

35 by creating the application header, transport head- 
er and frame relay header. The created frame is 
transmitted along with the packets of digital in- 
formation on the frame relay network, in step 126. 
A receiving codec operates in step 128 to 

40 remove the frame destined to the connected codec 
from the network node. The frame relay processor, 
in step 129, does the protocol layer processing, 
and removes the information packets for storage in 
an elastic buffer. The frame relay processor will 

45 perform the transport layer processing, i.e., se- 
quential checks, delay measurements, etc., in step 

130, as is common in frame relay protocol. 

Each of the voice, fax video applications run- 
ning in the background of the frame relay proces- 
so sor, obtains the frame from the transport layer, 
based on a digital signal processor request in step 

131. The removed frame is sent to the digital signal 
processor in step 132. This requires that the frame 
be retrieved from the elastic buffer and it be iden- 

55 tified as containing the appropriate media informa- 
tion for the requesting digital signal processor. 

The connected digital signal processor per- 
forms signal processing including voice decoding, 
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fax modulation or video/image decoding, depend- 
ing on the information transfer in step 133. The line 
interface unit on the outgoing side of the digital 
signal processor will, for appropriate media, make 
the digital-to-analog conversion for transmission to 
the connected terminal equipment. 

Thus, it can be seen that virtually all commu- 
nication media can be reduced to digital packets 
which can be transported via frame relay network 
to a connected codec. 

Following any one of these information trans- 
fers, a call disconnect is originated in which a user 
having completed the information transfer, will ini- 
tiate the disconnect in step 139. The receiving 
digital signal processor recognizes the disconnect 
request and, in the case of voice, goes on-hook, in 
the case of fax on-hook, or identifies to a video 
operator the end of transmission period. 

The application, running in the frame relay pro- 
cessor, will inform the sending codec of the dis- 
connection, via a subsequent issued frame over the 
frame relay network. The receiving digital signal 
processor will disconnect its line, going on-hook in 
the case of a voice or fax transmission, and in the 
case of video, a disable mode in step 144. The 
application sends a response to the remote ap- 
plication in step 1 45, indicating it has completed its 
disconnect. 

The multimedia codec will receive frames over 
the network pursuant to an information transfer 
which may be of any one of the media types. As 
the receipt of such frames must be handled on a 
priority basis, the frame relay processor operates in 
the foreground on received frames which are des- 
tined for a connected codec. Foreground execution 
begins from an interrupt issued to the frame relay 
processor from the network and stores each of the 
information fields received in a circular buffer, per- 
mitting applications running in the background of 
the frame relay processor to access this data and 
forward it out to an appropriate digital signal pro- 
cessor. 

Figure 11 illustrates the elastic buffer and its 
list of pointers which permit the background ap- 
plications access to received frame data. Figure 1 1 
illustrates a circular buffer wherein data is stored 
for applications X, Y and Z. These applications 
may be distinctly different media type such as 
telephone, fax and/or video. 

The frame relay layer receives frames from the 
frame relay network in the foreground, wherein an 
interrupt switches the frame relay processor from 
background to foreground operation. The received 
frames are "written in the circular buffer. At this 
time, the delay measurement variable in the trans- 
port header of the frame is read, and the measured 
delay value for the transport connection is updated. 
Each of the stored frames is identified in a list of 
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pointers, dedicated to each of the different applica- 
tion types serviced by the frame relay processor. 
Thus, voice frames are identified in one list of 
pointers, fax frames in another list of pointers, and 
5 video frames in another list of pointers, etc. The 
size of the elastic buffer is pre-allocated, based on 
the frame relay line rate and the application bit 
rates. 

The frame relay processor reverts to the back- 
70 ground mode, once the incoming frame has been 
stored. The transport and application layers pro- 
cess the frame in the background. If the transport 
header indicates a transport layer control message, 
the message is processed by the transport layer. If 
75 the transport header indicates information is 
present in the frame, then the transport layer 
checks with the transmit/receive sequence num- 
bers for error, assuming error recovery is required 
by the application. The pointer to the valid frame is 
20 written in the corresponding applications 1 list of 
pointers. 

The application layer reads and processes its 
receive frames, identified from the list of pointers. 
Frames are processed differently by the application 

25 layer.de pending on the application .In the case of 
voice, the voice application reads the application 
message type to determine whether this frame is a 
control message or voice information. 

If it is voice information, the transmit sequence 

30 number is compared with the next expected trans- 
mit sequence number to determine whether the 
voice frame is to be given to the respective digital 
signal processor, or deferred and instead silent 
frames are given to accommodate gaps in speech 

35 spurts. 

The application for processing fax frames 
reads the application message types and pro- 
cesses fax protocol messages. The image informa- 
tion contained in a frame is given to the digital 

40 signal processor for decoding modulation and 
transmitting out over the corresponding line inter- 
face unit to the connected fax equipment. 

The video application reads the application 
message type and processes any control message 

45 contained therein. Video data contained in the 
frame is given to its respective digital signal pro- 
cessor for decompression and transmission to the 
connected video terminal equipment. 

In the case of computer data received from the 

50 frame, a data application running in the background 
of the frame relay processor reads the received 
frames and outputs them transparently to a data 
port connected to computer terminal equipment. 
Each of the applications have a delay param- 

55 eter to determine when to start reading the elastic 
buffer. This will allow the application to accumulate 
a certain number of frames in the elastic buffer to 
provide for any cushion against the changes in 
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network delay. The delay parameter is controlled 
dynamically based on the network delay measure- 
ments done by the transport layer processing. 

Applications that send mixed media data, such 
as audio, fax and video in the same frame, for 
synchronization or bandwidth saving purposes, are 
handled the same. The transport layer for such 
frames determines that more than one application 
data is contained in the frame by checking the ID 
field, which will identify a group of applications 
being serviced by the frame. The transport layer 
writes the frame pointer in multiple lists of pointers 
for these applications. Each application then pro- 
cesses ,the frame independently, using only its 
portion of the information from the application in- 
formation, field of the frame. 

Thus, there has been described with respect to 
one embodiment a multimedia codec which may 
be implemented on frame relay network architec- 
ture. Those skilled in the art will recognize yet 
other embodiments described more particularly by 
the claims which follow. 

Claims 



2. A multimedia frame relay codec of claim 1 
further comprising a fourth application running 



in said frame processor background mode, 
which forms a data frame from data received 
from said digital signal processor, said data 
frame including a multimedia transport header 
s for identifying to a receiving codec connected 

to said frame relay common multimedia trans- 
port functions. 

3. The multimedia frame relay codec of claim 2 
w wherein said data frame includes a multimedia 

application header which identifies whether 
said frame includes voice data, video data or 
computer terminal data. 

75 4. The frames relay codec of claim 1, 2 or 3 
wherein said frame relay processor creates 
lists of pointers, each list identifying the buffer 
locations having respectively voice data, video 
data and computer data. 

20 

5. The frame relay codes of claim 1, 2, 3 or 4, 
wherein said background mode executes at 
least two applications which read said voice 
data, video data and computer data, respec- 

25 tively, and route said voice data, video data ad 

computer data to said digital signal processor. 

6. A multimedia frame relay codec comprising: 
first, second and third line interfaces con- 
nected to first, second and third different sour- 
ces of media; 

first, second and third digital signal pro- 
cessors connected to said first, second and 
third line interfaces; 

a frame relay processor connected to each 
of said digital signal processors and to a frame 
relay network said frame relay processor hav- 
ing an elastic buffer for storing incoming media 
data packets, said frame relay processor pro- 
grammed to operate in a foreground mode to 
store received data packets in said elastic buff- 
er, and create pointer lists identifying packets 
by media type, and said frame relay processor 
executing in the background applications for 
transferring said media data packets to respec- 
tive digital signal processors. 

7. The multimedia frame relay codec of claim 6, 
wherein said frame relay processor is pro- 
grammed to execute an application to create a 
frame of data for transmission over said frame 
relay to a receiving codec connected to said 
frame relay network. 

8. The multimedia codec of claim 7 wherein said 
frame of data includes an application field for 
identifying information packets contained in 
said frame as originating from said first, sec- 



1. A multimedia frame delay codec comprising: 

at least two of the following interface units; 

a voice line interface unit for providing 
digitized voice data packets; 

a video line interface for providing digitized 30 
video signal packets; 

a data line interface for providing data 
packets generated from a computer terminal; 

at least one digital signal processor, con- 
nected to said line interface units, for encoding 35 
signals received from a respective line unit and 
decoding signals received from a frame relay; 

a frame relay processor connected to said 
digital signal processor, said frame processor 
being programmed to operate in a foreground 40 
mode in response to receipt of an interrupt 
generated in response to receipt of a data 
frame from said frame relay, said processor 

during said foreground mode storing data 
fields comprising at least two of voice data, 45 
video data ad computer terminal data in an 
elastic buffer, and generating lists of pointers 
for identifying where said voice, video and 
computer terminal data packets respectively 
are stored; and, 50 

said background mode executing at least 
two applications which retrieve said stored 
voice data, video data and computer terminal 
data packets, respectively, and providing said 
data to a respective digital signal processor. 55 
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ond and third sources of media. 

9. In a multimedia frame relay codec having first, 
second and third line interfaces connected to 
first, second and third different sources of me- 5 
dia and at least one digital signal processor 
connected to said first second and third line 
interfaces, a method for routing packets com- 
prising: 

storing incoming media data packets re- w 
ceived from a frame relay network in an elastic 
buffer and creating pointer lists identifying 
packets by media type, in a foreground mode; 
and 

transferring said media data packets to 75 
respective digital signal processors in a back- 
ground mode. 

10. The method of claim 9 further comprising the 
step of generating an interrupt in response to 20 
the receipt of a packet from said frame relay 
network and wherein the steps of storing and 
created are performed in response to the inter- 
rupt. 

25 



30 



35 



40 



45 



50 



55 



10 



EP 0 669 742 A2 



21 



AUDIO 



M-LIU 



FAX 



22 



M-LIU 



IMAGET 



23 



M-LIU | 



VIDEO 



24 



M-LIU[- 



DATA 



25 



M-LIU 



FIG. 1 




APPLN ID (x) 



LIST OF 
POINTERS 



FRAME 1 S 



FRAME 2 



APPLN ID (y) 



LIST OF 
POINTERS 



I 



A 
//\ 

APPLN ID (z) | 

/ I 



LIST OF 
POINTERS 



I 



FIG.11 



CIRCULAR BUFFER 



APPLN ID (y) 



APPLN ID (x) 



APPLN ID (z) 



APPLN ID ( z) 



11 



EP 0 669 742 A2 



o 
m 

11 



H CD 

CC z 

Oocm 

Q- LU CO 
W> UJ 

z < o 
o 

£ oc 



a: o 

UJ > to 
o < co 
p: -< LU 

>D. 0 
< Q- 



Q. 
CO 
Q 



r 



in 



LU ? 
>CO 
<CO 

: lu 

< Ol 



UJ^ 

9z 

OCJ 

>co 



CD 

z 

CO 

co 

UJ 

o 
O 
oc 

Q_ 



CO, 
CO 



n 

CM 



r 



CO 

to 



DC O 
LU Z 

O >: co 

LU < CO 
Q — 1 LU 

>Q.Q 

< Q. 



C3 



O 
Z 

is 

co O 
cc 

Q_ 



E- 



UJ 
UJ 

o 



UJ 



3 
CO 

o 

I 



UJ 






X 















o 

O <! co 
uj 5 co 

> coo 

GC 

a. 



^9 




A/D 
-D/A 


00 / 
CO 



CO 




UJ 

o 

O 
> 




o 




UJ 1 


j. 


a ^ 


> 





CO 









AX 


§1 


ATA 






a 




> " 






12 



I * 

EP 0 669 742 A2 



< 

o < 

LU —J 
o 



UJ 

o 

§ 



Q. 
< 



CC 
HI 

a — ' 

w t— 
5 cc 

FO 

—I Q_ 
3 CO 

2 ? 

CC 



O 



UJ 
Z 



cc 

UJ -J 



LU 

< 

CC 



< cc 

LU 

< 

CC 



cc 

LU 



< 

o 

CO 

> 

X 
CL 



LU 
-J 

o 

LU 
Z 



< cr 

LU -» 



CC 
UJ 

<^ 



CC 

o 



z 



Q 
UJ 



11. = 



Q— 1 

I 



3 
2 



UJ 

o 

o 
> 



CL" 

OL 

< 



CC 

O 

Q. 
CO 

§ 



UJ 

LU 
CC 

LU 

5 
< 

CC 



DC 
UJ 

5 



LU 

CC 

UJ 

5 
< 

CC 



CC 
UJ 



CC 
LU 
> 



< 

o 

CO 

> 

X 

a. 



QC 
O 



UJ 

z 
>- 

£ CO 

LU 

£ LL 

UJ 

z 



CD 
Q 



< 



o 

CC 

o 



§1 

Si 

L O 

< s 



< 

Q DC 

5<£ 



ogee 

UJ O LU 

il 1 



LU 



- CC 
> UJ 



159 

2e3 



LU 
-J 
Q 



LU 
Z 



13 



( 'I 



EP 0 669 742 A2 



DSP 



FRP 



0/1 



SYNC FLAG 



FRAME RELAY HEADER 



TRANSPORT HEADER 



APPLICATION HEADER 



ENCODED 
DATA 



DSP-TO-FRP BUFFER 




DECODED 
DATA 



FRP-TO-DSP BUFFER 



FIG. 4B 
FIG. 5 

( START ) — 70 



I CRC I 



TX 
FRAME 



FRAME 
RELAY 



TRANSMISSION 
LINE 



RX 
FRAME 



CODEC POWER-UP —71 






FRP LOADS FROM SYSTEM DISK 







72 



FRP DOWNLOADS DSP(s) 
WITH VOICE/FAX/VIDEO SAV 
BASED ON CONFIGURATION 



— 73 



FRP-DSP INTERFACE INITIALIZED 












CODEC READY 


^ 75 



74 



5 



14 



EP 0 669 742 A2 



1 



80 

J 



DSP WAITS FOR OFF-HOOK ON PHONE LINE l/F 
OR FRAME RELAY INPUT 



81 



82 



OFF-HOOK 
(USER FAX CALL) 



83 

\ 



X 



86- 



r 



DSP SENDS DIAL-TONE 



DSP COLLECTS DIALLED DIGITS 
PASSES TO VOICE APPLN ON FRP 



VOICE APPLN MAPS DIGITS 
TO NETWORK ADDRESS & 

SETS UP NETWORK & 
TRANSPORT CONNECTIONS 



c 



84 
85 



APPLN WAITS FOR REMOTE 
PHONE LINE CONNECTION 



c 



FRAME REL 
(NETWORK V 


AY INPUT 
OICE CALL) 






GO OFF-HOOK, DIAL DIGITS 
WAIT FOR ANSWER 






SEND RESPONSE TO 
REMOTE VOICE APPLN 




\ 
88 



VOICE CONVERSATION 

FIG. 6A 



"V" 
90 



3 



15 



EP 0 669 742 A2 



WAIT FOR SAMPLE TIME TO 
RECEIVE DATA FROM LIU 



> 



ENCODE DATA 



— 92 



COPY ENCODED DATA TO 
DSP-TO-FRP BUFFER 



93 



SET SYNC FLAG = 1 



(5 



94 



WAIT FOR SYNC FLAG 



READ FRP-TO-DSP BUFFER 



95 



96 



DECODE DATA 



— 97 



TRANSMIT TO LIU 



98 



FIG. 6B 



16 



EP 0 669 742 A2 



(wait for sync flag =T)^, 1 



02 



READ DSP-TO-FRP BUFFER 



104- 



"103 



SILENCE h 



YES 



NO 



INSERT APPLICATION AND TRANSPORT 
HEADERS 



105 



ADD FRAME RELAY HEADER 



107 



•106 



TRANSMIT ON FRAME RELAY LINE 



108 



READ ELASTIC BUFFER 



109- 



SILENCE 



120 



YES 



NO 



110 



BUILD COMFORT 
NOISE FRAME 



WRITE FRAME INTO FRP-TO-DSP BUFFER 



SYNC FLAG = 0 



\ 
121 



FIG. 6C 



17 



EP 0 669 742 A2 



C 



130 



DSP WAITS FOR OFF-HOOK ON PHONE LINE l/F 
OR FRAME RELAY INPUT 



J 



131- 



132 



OFF-HOOK 
(USER FAX CALL) 



137v 



DSP SENDS DIAL-TONE 



133 



DSP COLLECTS DIALLED DIGITS 
PASSES TO FAX APPLN ON FRP 



FAX APPLN MAPS DIGITS 
TO NETWORK ADDRESS & 

SETS UP NETWORK & 
TRANSPORT CONNECTION 



FAX APPLN WAITS FOR REMOTE 
PHONE LINE SETUP 



134 



135 



FRAME REl 
(NETWORK 


.AY INPUT 
FAX CALL) 






GO OFF-HOOK, DIAL DIGITS 
WAIT FOR ANSWER 


^ 138 




SEND RESPONSE TO 
REMOTE FAX APPLN 


\ | 


\ 

139 

140 



FAX APPLN EXCHANGES TONES/ 
FACILITIES/TRAINING INFO 
WITH PEER APPLN LAYER, 
DSP PROVIDES DEMOD/MOD 



c 



IMAGE TRANSFER 



-y-14t 



® 

FIG. 7 



18 



EP 0 669 742 A2 



145 



2l 



3 



VIDEO APPLN WAITS FOR OPERATOR TO ENABLE VIDEO 
OR FRAME RELAY INPUT 



146 


1 


OPERATOR ENABLES VIDEO 


148 v 




DSP DETECTS LOCAL PORT UP 


149 v 





147. 



r 



FRAME RELAY INPUT 
(FROM REMOTE VIDEO 
APPLN) 



150 



1 



DSP DETECTS LOCAL 
PORT UP 



151 



VOICE APPLN SETS UP 
NETWORK & 
TRANSPORT CONNECTIONS 



c 



J 



A. 



SEND RESPONSE TO 
REMOTE VIDEO APPLN 



APPLN WAITS FOR REMOTE 
LINE CONNECTION 



c 



> 



155 



VOICE TRANSFER 



FIG. 8 



3 



19 



EP 0 669 742 A2 



160 





INFORMATION TRANSFER 




161 




168 >^ 





c 



INFO RECEIVED FROM 
MULTIMEDIA LINE 



162 



LIU PERFORMS ANALOG-TO-DIGITAL 
CONVERSION (IF NEEDED) 



163 



DSP PERFORMS SIGNAL 
PROCESSING 
VOICE: ENCODING 
FAX: DEMODULATION 
VIDEO/IMAGE: ENCODING 



164 



21. 



DSP PERFORMS ACTIVITY 
DETECTION AND PACKETIZATION, 
SENDS INFO TO FRP 



165 



FRP INSERTS 
APPLN HEADER, 
TRANSPORT HEADER, 
FRAME RELAY HEADER 



166^ 




FRP TRANSMH 
FRAME RE 


rS FRAME ON 
LAY LINE 



INFO RECEIVED FROM 
FRAME RELAY LINE 



69 



L 



FRP PERFORMS FRAME RELAY 
LAYER PROCESSING & PUTS 
FRAME IN ELASTIC BUFFER 



170 



FRP PERFORMS TRANSPORT 
LAYER PROCESSING (EG. SEQ 
CHECKS, DELAY M/S) 



171 
J— 



APPLN GETS FRAME FROM 
TRANSPORT LAYER (BASED ON 
DSP REQUEST) 





172 

/ 


APPLN SENDS FRAME TO DSP 




173 

\ 


DSP PERFORMS SIGNAL 
PROCESSING 
VOICE: ENCODING 
FAX: DEMODULATION 
VIDEO/IMAGE: DECODING 




174 

\ 


LIU PER 
DIGITAL-T( 
CONVEF 
SENDS SIGh 


FORMS 
>ANALOG 
3SION & 
JALON LINE 



1 



FIG. 9 



1 



20 



EP 0 669 742 A2 



180 



USER ORIGINATED 



181 

A. 



DSP RECEIVES DISCONNECT REQ: 
VOICE: ON-HOOK 
VIDEO: OPERATOR CMD 



182 



APPLN INFORMS PEER 
REMOTE APPLN 



APPLN DISCONNECTS 
TRANSPORT & NETWORK 
CONNECTION 



183 

I I— 

NETWORK ORIGINATED 
( FROM REMOTE APPLN 



D 



184 



DSP DISCONNECTS LINE: 
VOICE: ON-HOOK 
FAX: MSG/ON-HOOK 
VIDEO: DISABLE LINE 





185 

/ 


APPLN SEND 
TO REMO 


S RESPONSE 
TE APPLN 



FIG.10 



21 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 
OnSLACK borders 



□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 




DED TEXT OR DRAWING 



